In computer systems, a system clock signal is distributed throughout the system as a reference signal to control the timing of events. The use of a reference clock simplifies the design of data transfer structures within the computer since they can be designed to operate synchronous to the system clock. Synchronous designs are easier to conceptualize and to partition.
System clock signals are typically distributed from a single source point to various destination points within the computer system, which may be located some distance apart. In complex, high performance data processing systems such as those manufactured by Cray Research, Inc., the assignee of the present invention, the clock frequency is high and the number of points to which the system clock is provided is large. In art systems, high frequency clock signals were typically distributed as electrical signals sent point-to-point from the source to the destination. These electrical point-to-point distribution systems have several disadvantages. Point-to-point connections made in this manner act as individual antennae radiating radio frequency (RF) energy, adding to system noise and significantly affecting reliability. Also, signals distributed by this means do not arrive at all destination points at exactly the same time. The differences in time between these arrivals is called skew.
Within a computer system, data is passed from register to register, with varying amounts of processing performed between registers. Registers store data present at their inputs either at a system clock transition or during a phase of the system clock. Skew in the system clock signal impacts register-to-register transfers, i.e., it may cause a register to store data either before it has become valid or after it is no longer valid.
On slower computer systems, skew is usually a small portion of the clock period. Its impact can be reduced by adding delay to the data path. However, on high performance data processing systems such as supercomputers manufactured by Cray Research, Inc., the same amount of clock skew may be a substantial portion of the clock period and may actually limit the speed at which the computer system can operate. In addition, the amount of delay that can be added to a data path is limited. Therefore a skew becomes a major design factor in the transfer of data in high speed computing systems.
Clock skew is caused by a number of factors. A typical path for an electrical clock signal will include interconnections between circuit boards, fanout gates, circuit boards foil paths, and integrated circuit (IC) interconnect metal. Each of these interconnections provides an opportunity for introducing undesired clock skew.
Another source of clock skew is varying signal path lengths in the clock distribution network. The amount of time it takes a signal to travel along a wire, foil path, or interconnect metal is called its electrical path length and is dependent upon physical length and capacitance. All else being equal, a signal will take longer to travel a long path than a short one. If the electrical path lengths of all the clock signal paths are not equal, skew is introduced.
Another major source of clock skew is due to the integrated circuits that drive the clock signals. In a typical system the clock signal is distributed to a limited number of clock fanout devices on a circuit board. These clock fanout devices in turn distribute the clock signal to the remaining integrated circuits on the board. There will always be differences in propagation delay between fanout devices. Some reasons for these differences are variations in the semiconductor manufacturing process, temperature and voltage. Propagation delay differences between fanout devices further increases clock skew. In addition, in systems where there are a large number of integrated circuits, two or more levels of fanout devices may be needed to provide a clock signal to all of the registers in the system. Each level of fanout has the potential of adding additional skew to the system clock signal.
Crosstalk from adjacent signals, coupled RF interference and power and ground noise also act to increase clock skew in electrical clock distribution systems. For example, if a signal's voltage level is altered by crosstalk, the point in time when the signal is determined to have changed from a high to a low level will be altered, thus introducing clock skew.
As system clock periods shrink there is increasing pressure on the computer architect to reduce indeterminism in the system design. Clock skew, like setup time, hold time and propagation delay, increase the amount of time that data is in an indeterminable state. System designers must be careful that this indeterminable state does not fall within the sampling window of a register in order to preserve data integrity. In art systems, data path delay was often used to move the location of the indeterminable state outside the sampling window. However, as clock frequencies approach 500 MHz and go beyond, the ability to position the indeterminable state becomes more difficult. As a result clock skew reduces the ability to increase the system clock frequency.
Several techniques have been used in clock distribution networks in an attempt to reduce clock skew. System designers attempt to equalize the wire, foil path, and interconnect metal lengths between the clock source and all destinations. This helps to reduce variation in the electrical path length due to physical length. However, since each path may have a different impedance and/or capacitance, it is difficult to truly match electrical path length.
Methods are well known in the art for minimizing crosstalk, shielding signals and providing more stable temperature and voltage references. Likewise, variation in propagation delay between fanout devices can be controlled to a degree. This can be done by mandating stringent screening requirements or by matching components during assembly. However, the former drives up the price of the devices while the latter drives up the manufacturing costs. Again, truly matched performance is difficult to achieve because of the different loads and impedances faced by each device.
Clock distribution network tuning is a different approach at reducing the effects of clock skew. Delay is added selectively to clock paths in an attempt to equalize the delay through each of the paths. A representative tuning strategy is disclosed in the co-pending and commonly assigned patent application Ser. No. 07/465,947 filed Jan. 16, 1990 by Stephen E. Nelson et al. entitled "CLOCK DISTRIBUTION SYSTEM AND METHOD", now U.S. Pat. No. 5,258,660, which application is incorporated herein by reference. That application discloses the use of selectable delay paths to tune clock signal paths to compensate for clock skew.
Optical clock distribution networks have been proposed as a viable alternative to electrical clock distribution networks. Optical fiber as a transmission medium provides numerous advantages. Optical fiber provides a noise-free signal transmission environment, is resistant to electromagnetic interference, generates no electromagnetic interference which could initiate crosstalk, and supports very high transmission rates. However, in optical clock systems proposed to date, high frequency clocks with minimal skew have been difficult to achieve.
The effectiveness and practicality of the methods discussed above varies. It is clear that there has existed a long and unfilled need in the art for a clock distribution method and apparatus capable of reducing clock skew. The present invention solves these and other shortcomings of the techniques known in the art.
An additional shortcoming of existing electrical or optical network tuning schemes is that although the distribution network may be tuned to eliminate some degree of clock skew, the receiver and fanout electronics necessary to distribute the clock signals to individual circuit board components introduce additional skew which is not accounted for by tuning the distribution network. Because these tuning strategies do not tune all the way to the individual circuit board components, their ability to effectively minimize clock skew is significantly limited.
Because of the above discussed deficiencies, it is clear that there has existed a long and unfilled need in the art for a clock distribution system capable of reducing clock skew to the degree that system clock frequencies of 500 MHz to 1 GHz and beyond can be attained. The present invention solves these and other shortcomings of the techniques known in the art.